Hardware
Host Server srv.abair.ie
The server is setup as a Proxmox Hypervisor running Debian 12, the host system merely serves as a Jump Host for servers further into the network, and as an LDAP server other than being a hypervisor. The host Server IP Address is 136.243.40.234
AX102
The Abair Infrastructure is comprised of one Hetzner Dedicated Server, an AX102 as well as an additional 5TB storage block. The Hetzner account is under Rían Errity's email at the moment, but Ailbhe Ní Chasaide and John Sloan are setup as billing emails to ensure the invoices are well-received and paid for promptly.
Virtual Machines
The main purpose of the host machine is to host virtual machines which the actual services run on. This provides network and access isolation to the various teams of researchers who work on the production services in the servers' care.
They are full KVM/QEMU virtual machines, some of which use Host cores rather than KVM vCores for performance or CPU flag reasons (for instance, MongoDB v5 requires a CPU with AVX enabled)
A full list of virtual machines and containers on this host server can be found in Server List
Specs
| Key | Value |
|---|---|
| Product | Hetzner:AX102 |
| CPU | AMD Ryzen 9 7950X3D 16-Core Processor |
| Memory | 125GB 3600 MT/s |
| Disk | 2x1.75 TiB NVME SW Raid 1 |
| OS | Debian 12 (Proxmox VE) |
Proxmox
Proxmox VE is an open‑source virtualization platform based on Debian Linux that integrates the KVM hypervisor and LXC containers into a single, web‑based management interface. It provides built‑in clustering, software‑defined storage, high‑availability, and backup tools to simplify running and scaling virtual machines and containers.
Our Hetzner servers are managed through proxmox.
Backup Storage
Alongside our server space we're also renting a BX31 10TB storage box from Hetzner which provides us CIFS/SAMBA access to a 5TB redundant drive, this is mounted to /mnt/backup on the host system which serves as a Proxmox storage drive for VM Snapshots which are taken at 9pm nightly, with a 7-day retention period.
TODO: Backup proxmox configuration itself
Other Hardware
While we have several on-premises servers, these are currently out of scope of this documentation as they will need to be re-worked significantly to bring them into conformance with the rest of our network.
AMERGIN - GPU Server
This server is setup for serving streaming and batch recognition. It runs Debian 13. The host Server IP Address is 5.9.9.57
Specs
| Key | Value |
|---|---|
| Product | Hetzner:GEX44 |
| CPU | Intel Core i5-13500 (14-Core: 6 P-cores, 8 E-cores) |
| GPU | NVIDIA RTX 4000 SFF Ada Generation (20 GB GDDR6 ECC) |
| Memory | 64 GB DDR4 |
| Disk | 2x 1.92 TB Gen3 NVMe SSD (Datacenter Edition) |
| OS | Debian Trixie |
User
We use Ansible for automation. There is only one user: ansible-control
Setup Steps
Note: Putting these details here for now. Need to rearrange all Infrastructure docs soon
1. Create and Configure ansible-control User
First, log into the server as root to create the new user, grant administrative privileges, and copy over your SSH key.
ssh root@<SERVER_IP>
Create the new user:
adduser ansible-control
Add the user to the sudo group:
usermod -aG sudo ansible-control
Set up the SSH keys for the new user by copying root's authorized keys (user's machine must be the control node):
su - ansible-control
mkdir ~/.ssh
chmod 700 ~/.ssh
sudo cp /root/.ssh/authorized_keys ~/.ssh/
sudo chown -R ansible-control:ansible-control ~/.ssh/
chmod 600 ~/.ssh/authorized_keys
exit
Test the connection: Open a new terminal window on your local machine and test logging in with the new user before proceeding:
ssh ansible-control@<SERVER_IP>
2. Lock Down SSH
Run the remaining steps while logged in as ansible-control using sudo.
Restrict SSH access solely to the ansible-control user, disable root login, and enforce public key authentication.
sudo nano /etc/ssh/sshd_config
Find and modify (or add) the following settings:
PermitRootLogin no
PasswordAuthentication no
AllowUsers ansible-control
Check the config file has been correctly formatted:
sudo sshd -t
Restart the SSH service:
sudo systemctl restart ssh
(Verify you can still connect via a new terminal before closing the current one).
3. Configure UFW (Firewall)
Block all incoming traffic except SSH connections originating specifically from the Ansible control node IP.
sudo apt update
sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
Allow SSH only from the control node (replace <CONTROL_NODE_IP> with the actual IP). This is the IP from which you will be running ansible (e.g. your local machine / phoneticserv3):
sudo ufw allow from <CONTROL_NODE_IP> to any port 22
Enable the firewall:
sudo ufw enable
4. Configure Sudo
To allow Ansible to run commands without prompting for a password, configure passwordless sudo for the ansible-control user.
sudo visudo
Add this line to the bottom of the file:
ansible-control ALL=(ALL) NOPASSWD: ALL
5. Enable Automatic Security Updates
Since this server is managed automatically, configure Debian to install critical security patches without human intervention.
sudo apt install unattended-upgrades apt-listchanges
sudo dpkg-reconfigure -plow unattended-upgrades
Select Yes when prompted.
6. Install Fail2ban
Add a layer of defense against IP spoofing or brute-force attempts.
sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban